Image Projection Device

ABSTRACT

An image projection device which can correctly discern content of touch operation when a user performs various kinds of touch operation on an image projected on a projection screen is provided. An imaging unit is configured to image the image projected on the projection screen and acquire image data. A reference data generating unit is configured to generate reference data specifying a position and a size of the image projected on the projection. An image data extracting unit is configured to extract image data in which a finger or a pointer with which a user performs operation on the image projected on the projection screen exists is focused in the image data obtained by the imaging unit. A position data generating unit is configured to generate position data for specifying a position of the finger or the pointer in the imaging range of the imaging unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/072,436, filed Jul. 24, 2018, which is a 371 of InternationalApplication No. PCT/JP2016/082469, filed Nov. 1, 2016, which claimspriority to JP 2016-011926 filed Jan. 25, 2016, the disclosures of allof which are expressly incorporated by reference herein.

BACKGROUND AND SUMMARY OF THE INVENTION

Currently, various techniques for detecting touch operation when a userperforms the touch operation on an image projected on a projectionscreen are proposed regarding a projector (see, for example, Patentdocument 1. Further, see Background Art described in Patent document 1.)In the technique disclosed in Patent document 1, a projected image on aprojection screen is photographed with a camera provided in the vicinityof a projector, a difference between an original image of the projectedimage projected from the projector and the photographed image from thecamera is evaluated, an area ratio between a region of shadow of thefinger and a region of the finger in a touch region on the photographedimage corresponding to a touch region on the projected image isobtained, and, when the area ratio becomes less than a predeterminedvalue, it is determined that the hand has touched the touch region.

By the way, touch operation includes various kinds of operation such astap operation, double-tap operation, long-tap operation (long-tap), dragoperation, flick operation and scroll operation. With the techniquedisclosed in the above-described Patent document 1, there is a problemthat these various kinds of operation cannot be correctly determined.For example, the user touches an image twice with his/her finger so thatthe finger stays near the projection screen when the user performsdouble-tap operation. With the technique disclosed in theabove-described Patent document 1, because it is determined that thehand has touched the touch region when the area ratio of the region ofthe shadow of the finger and the region of the finger becomes less thanthe predetermined value, it is difficult to clearly distinguish betweena case where the finger has touched the image twice and a case where thefinger constantly touches the image, concerning double-tap operation.

The present invention has been made in view of the above-describedcircumstances, and is directed to providing an image projection devicewhich can correctly discern content of touch operation when a userperforms various kinds of touch operation on an image projected on aprojection screen.

To achieve the above-described object, an image projection deviceaccording to the present invention includes a projection screen, aprojection unit configured to project and display a predetermined imageon the projection screen, an imaging unit having a function of focusingand configured to image the image projected on the projection screen toacquire image data, a reference data generating unit configured togenerate reference data for specifying a position and a size of theimage projected on the projection screen in an imaging range of theimaging unit based on the image data obtained by the imaging unit, animage data extracting unit configured to extract image data in which afinger or a pointer with which a user performs operation on the imageprojected on the projection screen exists and in which the finger or thepointer is brought into focus, in the image data obtained by the imagingunit, a position data generating unit configured to generate positiondata for specifying a position of the finger or the pointer in animaging range of the imaging unit based on the image data extracted bythe image data extracting unit, an operation determining unit configuredto determine content of operation performed with the finger or thepointer based on the image data extracted by the image data extractingunit, and an input control unit configured to recognize content of aninput instruction corresponding to the operation performed with thefinger or the pointer based on the content of the operation determinedby the operation determining unit, the position data generated by theposition data generating unit, and the reference data generated by thereference data generating unit, and control the projection unit toproject an image in accordance with the recognized content of the inputinstruction on the projection screen. Here, it is preferable that theimaging unit is adjusted to come into focus within a range of apredetermined distance in a front-back direction from the projectionscreen along a vertical direction of the projection screen.

In the image projection device of the present invention, the imagingunit has a function of focusing, and the image data extracting unitextracts image data in which the finger or the pointer with which theuser performs operation on the image projected on the projection screenexists and in which the finger or the pointer is brought into focus, inthe image data obtained by the imaging unit. Therefore, for example, byadjusting the imaging unit so as to come into focus within a range of afixed short distance in a front-back direction from the projectionscreen along the vertical direction of the projection screen, becausethe image data extracted by the image data extracting unit includes onlyimage data in which the finger or the pointer exists in the vicinity ofthe projection screen, the operation determining unit can correctlydiscern content of operation by determining the content of the operationperformed with the finger or the pointer based on the image dataextracted by the image data extracting unit. The input control unit thenrecognizes content of the input instruction corresponding to theoperation performed with the finger or the pointer based on the contentof the operation performed with the finger or the pointer obtained bybeing determined by the operation determining unit, the position data ofthe finger or the pointer generated by the position data generating unitand the reference data generated by the reference data generating unit,and controls the projection unit to project an image in accordance withthe recognized content of the input instruction on the projectionscreen. Therefore, when the user performs various kinds of touchoperation on the image projected on the projection screen, the imageprojection device of the present invention can correctly recognize theinput instruction in accordance with the touch operation.

Further, in the image projection device according to the presentinvention, a frame may be pictured in each image projected on theprojection screen or predetermined marks may be pictured at four cornersof each image projected on the projection screen, and the reference datagenerating unit may recognize a position of the frame or the marks ofthe image in the imaging range of the imaging unit based on the imagedata obtained by the imaging unit and set data relating to therecognized position of the frame or the marks as reference data. By theimage having the frame or the marks being projected on the projectionscreen in this manner, the reference data generating unit can easilygenerate reference data relating to a position and a size of the image.

Further, in the image projection device according to the presentinvention, while the projection screen displays an image by projectionlight being irradiated from a back side of the projection screen, theprojection unit may project the image from the back side of theprojection screen, and the imaging unit may image the image from theback side of the projection screen to acquire image data. In this case,it is preferable that, when the user performs operation on the imagefrom a surface side of the projection screen with a finger or a pointer,the image data extracting unit recognizes a shape, or a shape and colorcorresponding to the finger or the pointer in the image data capturedthrough the projection screen and acquires image data in which therecognized shape or the recognized shape and color exists as image datain which the finger or the pointer with which the user performsoperation on the image exists. Further, in the image projection deviceaccording to the present invention, the projection screen may display animage by projection light being irradiated from a surface side of theprojection screen, in which case, the projection unit projects the imagefrom the surface side of the projection screen, and the imaging unitimages the image from the surface side of the projection screen toacquire image data.

Further, in the image projection device according to the presentinvention, in the case where a laser pointer which emits laser light isused as the pointer, when the user performs operation on the imageprojected on the projection screen by irradiating laser light from thelaser pointer on the projection screen, the image data extracting unitmay recognize a shape and/or color of the laser light from the laserpointer in the image data obtained by the imaging unit and acquire imagedata in which the recognized shape and/or color of the laser lightexists as image data in which the finger or the pointer with which theuser performs operation on the image exists. By this means, also in thecase where the laser pointer is used as the pointer, the imageprojection device can correctly recognize the input instruction from theuser.

Further, it is preferable that the image projection device according tothe present invention includes a table on which a screen is placed forattaching the projection screen so that a distance between theprojection screen and the projection unit and a distance between theprojection screen and the imaging unit are respectively maintained atsubstantially fixed distances. As a result of such a table on which thescreen is placed being used, the user can easily set positionalrelationship between the projection screen and the projection unit andpositional relationship between the projection screen and the imagingunit.

According to the image projection device according to the presentinvention, when a user performs various kinds of touch operation on animage projected on a projection screen, it is possible to correctlyrecognize an input instruction in accordance with the touch operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic configuration diagram of a computer terminalincluding an image projection device which is an embodiment of thepresent invention.

FIG. 2 is a schematic block diagram of the computer terminal includingthe image projection device of the present embodiment.

FIG. 3A is a schematic perspective view of a table on which a screen isplaced, FIG. 3B is a schematic side view illustrating how the table onwhich the screen is placed is folded, and FIG. 3C is a schematic planview of the table on which the screen is placed, which is in a foldedstate.

FIG. 4 is a diagram illustrating an example of an image of a characterinput screen.

FIG. 5 is a diagram schematically illustrating a portion of a fingerwhose shape or color is changed when a user brings the finger intocontact with a projection screen.

FIG. 6 is a diagram illustrating an example of an image of the characterinput screen in which, when the user performs tap operation on acharacter key image, the character key image is highlighted.

FIG. 7 is a flowchart for explaining procedure of input processing inthe image projection device according to the present embodiment.

FIG. 8 is a schematic configuration diagram of a computer terminalincluding a front-projection type image projection device which is amodified example of the present invention.

FIG. 9A is a schematic perspective view of a smartphone in which theimage projection device of the present invention is provided and FIG. 9Bis a schematic side view of the smartphone.

FIG. 10A is a schematic perspective view of a wristwatch in which theimage projection device of the present invention is provided and FIG.10B is a diagram illustrating how a projection screen is provided at thewristwatch.

FIG. 11A is a diagram illustrating an example of a car in which arear-projection type image projection device of the present invention isprovided and FIG. 11B is a diagram illustrating an example of a car inwhich a front-projection type image projection device of the presentinvention is provided.

FIG. 12 is a diagram illustrating an example of a desk in which theimage projection device of the present invention is provided.

FIG. 13A is a schematic perspective view of a compact mirror.

FIG. 13B is a schematic side view of a compact mirror.

FIG. 13C is a schematic perspective view of a compact mirror when animage is projected.

FIG. 13D is a schematic side view of a compact mirror when an image isprojected.

DETAILED DESCRIPTION

An embodiment for implementing the invention according to the presentapplication will be described below with reference to the drawings. FIG.1 is a schematic configuration diagram of a computer terminal includingan image projection device which is an embodiment of the presentinvention, and FIG. 2 is a schematic block diagram of the computerterminal including the image projection device of the presentembodiment.

The image projection device of the present invention recognizes contentof an input instruction corresponding to touch operation when a userperforms the touch operation on an image projected on a projectionscreen using a finger of the user or a pointer, and projects an image inaccordance with the recognized content of the input instruction on theprojection screen. Particularly, in the present embodiment, a case wherethis image projection device is incorporated into a computer terminalwill be described. As illustrated in FIG. 1 and FIG. 2 , the computerterminal 1 includes a computer body 10, a liquid crystal display unit20, a touch panel 30 and the image projection device 100 of the presentembodiment. The liquid crystal display unit 20 is provided on an upperface of a housing of the computer body 10. The touch panel 30 isprovided on a screen of the liquid crystal display unit 20. At the touchpanel 30, a position detecting unit (which is not illustrated) isprovided which detects a position of contact when contact operation isperformed on the screen of the liquid crystal display unit 20 andoutputs contact position information indicating the detected position tothe computer body 10. On the screen of the liquid crystal display unit20, various kinds of screens such as a menu screen, an applicationscreen and a character input screen are displayed. Note that a (touchpanel) display other than a liquid crystal display may be used in placeof the liquid crystal display unit 20.

The image projection device 100 of the present embodiment plays roles asa display apparatus and an input apparatus at the computer terminal 1.Here, as the image projection device 100, a rear-projection type imageprojection device which projects an image from a back side of aprojection screen is used. As illustrated in FIG. 1 and FIG. 2 , theimage projection device 100 includes a projection screen 110, aprojection unit 120, an imaging unit 130, a control unit 140, a storageunit 150, and a table 160 on which a screen is placed. The projectionunit 120, the imaging unit 130, the control unit 140 and the storageunit 150 are provided inside a housing of the computer body 10. Thecontrol unit 140 which controls the image projection device 100, in thepresent embodiment, also controls the whole computer terminal 1. Thatis, the control unit 140 also plays a role as a control unit of thecomputer body 10. Note that a control unit which controls the imageprojection device 100 and a control unit of the computer body 10 may beseparately provided.

As the projection screen 110, a transmission type screen, that is, arear-projection screen for displaying an image by projection light beingirradiated from a back side of the screen is used. This projectionscreen 110 is a hard-type single sheet screen, and is fixed at the table160 on which the screen is placed. In the present embodiment, as thetable 160 on which the screen is placed, a folding table is used. FIG.3A is a schematic perspective view of the table 160 on which the screenis placed, FIG. 3B is a schematic side view illustrating how the table160 on which the screen is placed is folded, and FIG. 3C is a schematicplan view of the table 160 on which the screen is placed, which is in afolded state.

The table 160 on which the screen is placed is provided for placing theprojection screen 110 and the computer body 10 so that the projectionscreen 110 and the computer body 10 are disposed with predeterminedpositional relationship. As illustrated in FIG. 3 , the table 160 onwhich the screen is placed includes two base portions 161 a, 161 b andtwo pole portions 162, 162. Here, FIG. 3 illustrates the table 160 onwhich only the projection screen 110 is provided. Each of the baseportions 161 a and 161 b is a plate-like member in a substantiallyrectangular shape, and the two base portions 161 a and 161 b areconnected so as to be able to be folded. Further, at one base portion161 a, at an end portion on a side opposite to a side on which the otherbase portion 161 b is connected, the two pole portions 162, 162 areattached. Each of the pole portions 162, 162 is a rod-like member, andgrooves are formed along respective central axes of the pole portions162, 162 on facing side surfaces. The projection screen 110 is attachedto the table 160 on which the screen is placed by being inserted betweenthe grooves of the two pole portions 162, 162. Further, at the otherbase portion 161 b, as illustrated in FIG. 1 , the computer body 10 isplaced. In the present embodiment, as illustrated in FIG. 3A, a concaveportion 163 is formed at a predetermined portion on a surface of thebase portion 161 b. This concave portion 163 is utilized as a markindicating a location where the computer body 10 is provided. Therefore,the user can easily set positional relationship between the projectionscreen 110 and the computer body 10 so that a distance between theprojection screen 110 and the projection unit 120 and a distance betweenthe projection screen 110 and the imaging unit 130 respectively becomesubstantially fixed distances only by attaching the projection screen110 between the two pole portions 162, 162 and disposing the computerbody 10 on the concave portion 163 of the base portion 161 b. Further,it is possible to make the table 160 on which the screen is placed flatas illustrated in FIG. 3C by folding the two base portions 161 a and 161b as illustrated in FIG. 3B.

The projection unit 120 projects and displays an image of apredetermined screen on the projection screen 110 from the back side ofthe projection screen 110. In the present embodiment, the projectionscreen 110 plays a role as a display apparatus of the computer terminal1. Therefore, on the projection screen 110, images regarding variouskinds of screens such as a menu screen and an operation screen on whichvarious kinds of icons are displayed, and a character input screen forinputting characters are displayed. FIG. 4 is a diagram illustrating anexample of an image of the character input screen. As illustrated inFIG. 4 , the character input screen 200 has a keyboard image 210 and adisplay region 220 for displaying input characters, or the like. In thekeyboard image 210, a plurality of character key images associated withrespective characters (including symbols), and a plurality of functionkey images to which specific functions are provided, are provided.Further, in an image of each screen projected on the projection screen110 by the projection unit 120, a frame is pictured. For example, theimage of the character input screen 200 illustrated in FIG. 4 includes arectangular outer frame 230 enclosing the keyboard image 210 and thedisplay region 220.

In the present embodiment, the user can give various kinds ofinstructions to the control unit 140 by performing predeterminedoperation on the image of the screen projected on the projection screen110 with a finger or a pointer. Here, as the pointer, for example, apointing stick is used. Specifically, the user gives an instruction byperforming predetermined operation while bringing the finger or thepointer into contact with the image of the screen projected on theprojection screen 110 from the surface side of the projection screen 110or performing predetermined operation in the vicinity of the image ofthe screen projected on the projection screen 110 from the surface sideof the projection screen 110 without bringing the finger or the pointerinto contact with the image. The control unit 140 then recognizescontent of the instruction and controls the projection unit 120 toproject the image of the screen in accordance with the recognizedcontent of the instruction on the projection screen 110. Here, theoperation performed on the image of the screen projected on theprojection screen 110 includes various kinds of operation such as touchoperation performed on a normal touch panel, that is, tap operation,double-tap operation, long-tap operation (long tap), drag operation,flick operation, scroll operation and the like. Note that each operationin the present embodiment described here refers to operationcorresponding to touch operation performed on a normal touch panelregardless of whether or not operation is performed while the finger, orthe like, is brought into contact with the image.

The imaging unit 130 which images an image projected on the projectionscreen 110 to acquire image data, as illustrated in FIG. 2 , includes acamera unit 131, an image processing unit 132 and a camera control unit133. The camera unit 131 has a lens and an image sensor. As illustratedin FIG. 1 , the camera unit 131 images the projection screen 110 fromthe back side of the projection screen 110. A range (imaging range)which can be imaged by the camera unit 131 is the whole projectionscreen 110. Further, in the present embodiment, the camera unit 131 isdirected to imaging the finger or the pointer with which the userperforms operation on the image projected on the projection screen 110,through the transmission type projection screen 110. The imageprocessing unit 132 performs processing of correcting color or gradationof the image of the image data captured by the camera unit 131 orperforms image processing such as compression of the image data. Thecamera control unit 133 controls the image processing unit 132 orcontrols exchange of image data with the control unit 140. Note that, inthe present embodiment, a case will be described where the imageprocessing unit 132 is provided at the imaging unit 130, but the imageprocessing unit 132 may be provided at the control unit 140 instead ofbeing provided at the imaging unit 130.

Further, in the present embodiment, the imaging unit 130 has a functionof focusing. Therefore, the camera control unit 133 includes anautofocus control unit 133 a. The autofocus control unit 133 a adjusts(controls) the camera unit 131 so as to bring the projection screen 110into focus. Specifically, in the present embodiment, because a distancebetween the projection screen 110 and the camera unit 131 is set inadvance at a substantially fixed value, the autofocus control unit 133 aperforms focusing processing so that a position away from the cameraunit 131 by the above-described predetermined distance is brought intofocus when imaging operation is started. Further, at this time, theautofocus control unit 133 a adjusts the camera unit 131 so as to comeinto focus within a range of a predetermined distance in a front-backdirection from the projection screen 110 along a vertical direction ofthe projection screen 110. The range (depth of field) in which thecamera unit 131 comes into focus is limited to a narrow range. Forexample, in the present embodiment, a case will be considered where theautofocus control unit 133 a adjusts the camera unit 131 so as to comeinto focus within a range of 10 mm in a front-back direction from theprojection screen 110 along the vertical direction of the projectionscreen 110. Therefore, while the imaging unit 130 performs imaging in astate where a subject (the finger or the pointer) is brought into focusif the subject with which operation is performed on the image of thescreen projected on the projection screen 110 is located in the vicinityof the projection screen 110, the imaging unit 130 cannot performimaging in a state where the subject is brought into focus if thesubject is located farther from the projection screen 110 than the depthof field. Note that, as an autofocus system, one of an active autofocussystem in which a subject is irradiated with infrared light, ultrasound,or the like and a distance is detected from time required for thereflected wave to return and an irradiation angle, and a passiveautofocus system such as a phase difference detection system or acontrast detection system in which a distance is measured by utilizingan image captured with a lens of the camera unit 131, may be used.

The image data obtained through imaging by the imaging unit 130 istransmitted to the control unit 140 and stored in the storage unit 150by the control unit 140. Further, the imaging unit 130 has a function ofcapturing a still image and a function of capturing a moving image, andthe control unit 140 can acquire still image data or moving image data,if necessary, as the image data.

The control unit 140 controls the whole image projection device 100. Forexample, the control unit 140 controls the projection unit 120 toproject the screen on the projection screen 110 or controls imaging bythe imaging unit 130. Specifically, as illustrated in FIG. 2, thecontrol unit 140 includes a display control unit 141, a reference datagenerating unit 142, an image data extracting unit 143, a position datagenerating unit 144, an operation determining unit 145 and an inputcontrol unit 146.

The display control unit 141 controls the liquid crystal display unit 20and the projection unit 120. Specifically, when the user performs touchoperation at the liquid crystal display unit 20, the display controlunit 141 recognizes content of an instruction by the touch operationbased on contact position information transmitted from the touch panel30. At this time, if the recognized content of the instruction is aninstruction for displaying a predetermined screen at the liquid crystaldisplay unit 20, the display control unit 141 controls the liquidcrystal display unit 20 to display the screen at the liquid crystaldisplay unit 20. Further, if the recognized content of the instructionis an instruction for displaying an image of a predetermined screen onthe projection screen 110, the display control unit 141 controls theprojection unit 120 to display the image of the screen on the projectionscreen 110. Further, when the projection unit 120 starts processing ofprojecting an image, the display control unit 141 performs processing ofadjusting the image so that the image is clearly displayed on theprojection screen 110. When an image is displayed on the projectionscreen 110, the display control unit 141 may control the liquid crystaldisplay unit 20 so that the same image as the image displayed on theprojection screen 110 is displayed at the liquid crystal display unit20.

The reference data generating unit 142 generates reference data relatingto the image projected on the projection screen 110 based on the imagedata obtained by the imaging unit 130. This reference data is data forspecifying a position and a size of the image in the imaging range. Forexample, in the case where the image of the screen illustrated in FIG. 4is projected on the projection screen 110, position data at four cornersof the outer frame 230 of the screen can be used as the reference data.Specifically, in the present embodiment, an XY coordinate system inwhich a horizontal direction is an X axis direction and a verticaldirection is a Y axis direction is set within the imaging range of theimaging unit 130. The reference data generating unit 142 recognizes aposition of the outer frame 230 of the screen based on the image dataobtained by the imaging unit 130 and acquires data relating to therecognized position of the outer frame 230, that is, XY coordinates ofthe four corners of the outer frame 230 in the XY coordinate system, asthe reference data. The reference data generated at the reference datagenerating unit 142 is stored in the storage unit 150. In this manner,in the present embodiment, by the image having the outer frame beingprojected on the projection screen 110, the reference data generatingunit 142 can easily generate reference data relating to the position andthe size of the image.

The image data extracting unit 143 extracts image data in which thefinger or the pointer with which the user performs operation on theimage projected on the projection screen 110 exists and in which thefinger or the pointer is brought into focus in the image data obtainedby the imaging unit 130. Here, whether or not the finger or the pointerexists is judged using a typical image recognition method. Therefore,the image data extracted by the image data extracting unit 143 includesonly image data in which the finger or the pointer is located in thevicinity of the projection screen 110. The image data extracted in thismanner is stored in the storage unit 150. Further, the image dataextracting unit 143 generates data relating to time (time data) at whichthe image data is captured for each piece of extracted image data andstores the generated time data in the storage unit 150 in associationwith the image data. At the operation determining unit 145 and theposition data generating unit 144, processing is performed based on theimage data extracted by the image data extracting unit 143.

Specifically, in the present embodiment, when the user performsoperation on the image from the surface side of the projection screen110 with the finger or the pointer, the image data extracting unit 143can recognize a shape, or a shape and color corresponding to the fingeror the pointer in the image data captured through the projection screen110 and acquire image data in which the recognized shape or therecognized shape and color exists as the image data in which the fingeror the pointer with which the user performs operation on the imageexists. Further, when the finger or the pointer contacts the projectionscreen 110, in the case where the shape or color of the finger or thepointer changes on the projection screen 110, the image data extractingunit 143 may recognize the changed shape or color and judge existence ofthe finger or the pointer. That is, when the user performs operation onthe image while bringing the finger or the pointer into contact with theimage from the surface side of the projection screen 110, the image dataextracting unit 143 may recognize change of the shape, change of color,or change of the shape and color of the finger or the pointer when theuser brings the finger or the pointer into contact with the image in theimage data captured through the projection screen 110 and acquire imagedata in which the recognized change of the shape, the recognized changeof color, or the recognized change of the shape and color of the fingeror the pointer exists as the image data in which the finger or thepointer with which the user performs operation on the image exists. Inthis case, for example, as the pointer, one whose tip is formed with asoft material and whose tip easily deforms when contacts the projectionscreen 110, can be used. Further, as an example, FIG. 5 schematicallyillustrates a portion of the finger whose shape or color changes whenthe user brings the finger into contact with the projection screen 110.In the case where the shape or color of the finger changes by the fingerbeing brought into contact with the projection screen 110, the imagedata extracting unit 143 can judge that the finger exists by recognizingthe changed shape or color of the portion of the finger.

Note that it is preferable that a figure such as a shape of a star or across shape is pictured or a sticker, or the like, on which such afigure is pictured is put at a tip portion of the finger (for example, afinger pad) or the pointer used by the user for operation. Further, itis also possible to form a tip portion of the pointer in a shape of theabove-described figure. In this case, the image data extracting unit 143judges whether or not the figure exists in the image data and judgesthat the finger or the pointer exists if the figure exists. Becauserecognition of the above-described figure is easier than recognition ofthe finger, or the like, itself, it is possible to easily and quicklyperform processing for judging existence of the finger, or the like.

By the way, as the rear-projection screen, there is a translucentprojection screen, an opalescent projection screen, or the like, as wellas a transparent projection screen. In the case where a translucentprojection screen, an opalescent projection screen, or the like, isused, it is difficult to judge whether or not the finger or the pointeris brought into focus. To address this problem, for example, thefollowing method can be used. That is, in the case where various kindsof projection screens such as a translucent projection screen and anopalescent projection screen are used, at least one image in which thefinger or the pointer contacts the projection screen is acquired inadvance, and the image is preset as an image for which it should bejudged that the finger or the pointer is brought into focus. The imagedata extracting unit 143 can correctly judge whether or not the fingeror the pointer is brought into focus by comparing image data which isactually captured with the preset image.

Further, the image data extracting unit 143 may perform processing ofexcluding data concerning an image which is currently projected on theprojection screen 110 on the image data obtained by the imaging unit 130and judge whether the finger or the pointer exists based on the imagedata subjected to the processing. By this means, it is possible toperform processing of judging existence of the finger, or the like, moreeasily and quickly.

The position data generating unit 144 generates position data forspecifying a position of the finger or the pointer in the imaging rangeof the imaging unit 130 based on the image data extracted by the imagedata extracting unit 143. Specifically, the position data generatingunit 144 acquires an XY coordinate of a position where the finger or thepointer exists (for example, a center position of the tip of the fingeror a center position of the tip portion of the pointer) in the XYcoordinate system set within the imaging range as position data. Theacquired position data is stored in the storage unit 150 in associationwith the image data.

The operation determining unit 145 determines content of operationperformed with the finger or the pointer based on the image dataextracted by the image data extracting unit 143. Here, as describedabove, the image data extracted by the image data extracting unit 143includes only image data in which the finger or the pointer exists inthe vicinity of the projection screen 110. Specifically, the operationdetermining unit 145 determines content of operation by examiningmovement of the finger or the pointer based on the position data and thetime data associated with each piece of image data for a series ofpieces of image data obtained in chronological order.

For example, in the case where image data in which the position of thefinger or the pointer remains virtually unchanged is extracted only fora certain short period, the operation determining unit 145 determinesthat tap operation is performed with the finger or the pointer. Further,in the case where, after image data in which the position of the fingeror the pointer remains virtually unchanged is extracted for a certainshort period, image data in which the finger or the pointer exists atsubstantially the same position is extracted for a certain short periodagain before a certain period of time has elapsed, it is determined thatdouble-tap operation is performed with the finger or the pointer. Stillfurther, in the case where image data in which the position of thefinger or the pointer remains virtually unchanged is extracted for equalto or longer than a certain period of time, it is determined thatlong-tap operation is performed with the finger or the pointer. Further,in the case where image data in which the position of the finger or thepointer remains virtually unchanged is extracted for a short period,and, subsequently, image data in which the position of the finger or thepointer sequentially changes is extracted, it is determined that dragoperation is performed with the finger or the pointer. In this manner,the operation determining unit 145 can correctly recognize content ofthe operation performed with the finger or the pointer as operationamong various kinds of operation such as tap operation, double-tapoperation, long-tap operation and drag operation by determining contentof the operation performed with the finger or the pointer based on theimage data extracted by the image data extracting unit 143. Datarelating to the recognized content of the operation performed with thefinger or the pointer is stored in the storage unit 150.

Further, in the present embodiment, because the imaging unit 130 isadjusted so as to come into focus within a range of 10 mm in afront-back direction from the projection screen 110 along the verticaldirection of the projection screen 110, the user can perform operationsuch as tap operation only by putting the finger or the pointer at aposition in a distance of approximately 10 mm from the projection screen110 without actually bringing the finger or the pointer into contactwith the projection screen 110. Because the image data extracting unit143 extracts only image data in which the finger or the pointer existswithin a range of approximately 10 mm from the projection screen 110,the operation determining unit 145 can correctly discern content of theoperation by determining the content of the operation performed with thefinger or the pointer based on the image data extracted by the imagedata extracting unit 143. Note that, as described above, in the casewhere the shape or color of the finger or the pointer changes on theprojection screen 110 if the finger or the pointer contacts theprojection screen 110, the image data extracting unit 143 can recognizechange of the shape or color upon contact and extract image data inwhich the recognized change of the shape or color exists. In this case,because the user performs operation while bringing the finger or thepointer into contact with the projection screen 110, the user canperform operation as if the user operated an existing touch panel, andthe operation determining unit 145 can correctly discern content of theoperation by determining the content of the operation performed with thefinger or the pointer based on the image data extracted by the imagedata extracting unit 143.

The input control unit 146 recognizes content of an input instructioncorresponding to the operation performed with the finger or the pointerbased on the data relating to the content of the operation determined bythe operation determining unit 145, the position data generated by theposition data generating unit 144 and the reference data generated bythe reference data generating unit 142, and controls the projection unit120 to project an image in accordance with the recognized content of theinput instruction on the projection screen 110.

For example, in the case where the image projected on the projectionscreen 110 is an image of a menu screen, the input control unit 146 canrecognize a range in which the menu screen exists in the imaging rangeof the imaging unit 130 based on the reference data relating to theimage of the screen. Further, because the input control unit 146 knows aconfiguration of the menu screen in advance, the input control unit 146can recognize a position and a range of each icon in the menu screen.Therefore, for example, in the case where the user performs double-tapoperation of an icon on the menu screen with the finger, the inputcontrol unit 146 can specify an icon which is an operation target byexamining a region of which icon in the menu screen, the position of thefinger obtained from the position data of the finger corresponds to, andcan specify that the operation is double-tap operation by examiningcontent of the operation. As a result, the input control unit 146recognizes that an input instruction corresponding to the operation ofthis time is an input instruction for displaying an image of a screenrelating to the operated icon and performs processing of displaying theimage of the screen relating to the icon on the projection screen 110.

Further, in the case where the image projected on the projection screen110 is an image of the character input screen 200 illustrated in FIG. 4, the input control unit 146 can recognize a range in which thecharacter input screen 200 exists in the imaging range of the imagingunit 130 based on reference data relating to the image of the screen.Further, because the input control unit 146 knows a configuration of thecharacter input screen 200 in advance, the input control unit 146 canrecognize a range of the keyboard image 210, a region of each characterkey image, or the like, in the character input screen 200. Therefore,for example, in the case where the user performs tap operation of acharacter key image on the keyboard image 210 with the finger, the inputcontrol unit 146 can specify the operated character key by examining aregion of which character key image in the keyboard image 210, theposition of the finger obtained from the position data of the fingercorresponds to, and can specify that the operation is tap operation byexamining content of the operation. As a result, the input control unit146 recognizes that an input instruction corresponding to the operationof this time is an input instruction of a key corresponding to theabove-described operated character key and performs processing ofdisplaying an image of a character for which an input instruction isgiven in the display region 220.

Note that, in the case where the user performs tap operation of acharacter key image and it is recognized that there is an inputinstruction of a character corresponding to the operated character key,the input control unit 146 may perform processing of displaying an imageof a screen in which only the operated character key image ishighlighted in the character input screen 200 on the projection screen110. By this means, the user can easily confirm the character keyoperated by the user. FIG. 6 illustrates an example of an image of thecharacter input screen 200 in which, when the user performs tapoperation on a character key image, the character key image ishighlighted.

In the storage unit 150, various kinds of programs, data, or the like,are stored. The programs stored in the storage unit 150 include, forexample, a screen display processing program for performing processingof screen display such as switching of a screen when operation ofselecting an icon is performed in the menu screen, a character inputprocessing program for performing processing of character input whenoperation of selecting a character key is performed in the characterinput screen 200, and the like. Further, the data stored in the storageunit 150 includes, for example, image data for various kinds of screens.Still further, the storage unit 150 is also used as a working memory.

Input processing in the image projection device 100 of the presentembodiment will be described next. FIG. 7 is a flowchart for explainingprocedure of input processing in the image projection device 100 of thepresent embodiment.

The user, for example, operates the touch panel 30 to give aninstruction of displaying an image of a predetermined screen on theprojection screen 110. When the control unit 140 (display control unit141) receives this instruction, the control unit 140 controls theprojection unit 120 so as to display the image of the screen on theprojection screen 110 (51). When projection of the image is started inthis manner, the control unit 140 starts imaging operation by theimaging unit 130 (S2).

The control unit 140 then causes the camera control unit 133 to executefocusing processing (S3). Specifically, in the present embodiment,because a distance between the projection screen 110 and the camera unit131 is set in advance at a substantially fixed value, the autofocuscontrol unit 133 a of the camera control unit 133 controls the cameraunit 131 so that a position away from the camera unit 131 by thedistance determined in advance is brought into focus. Further, thefocusing processing can be performed using other methods. For example,by infrared reflection coating being applied on the projection screen110 itself, by a frame being provided at the projection screen 110 or byreflecting portions being provided at four corners of the projectionscreen 110, the camera control unit 133 can detect a distance to theprojection screen 110 with an autofocus function and control the cameraunit 131 so that a position away from the camera unit 131 by thedetected distance is brought into focus. Note that it is also possibleto provide reflecting portions on the table 160 on which the screen isplaced instead of providing the reflecting portions at the projectionscreen 110.

Further, at this time, the display control unit 141 controls theprojection unit 120 to perform projected image adjustment processing sothat the image is clearly displayed on the projection screen 110 (S4).Specifically, in the projected image adjustment processing, the displaycontrol unit 141 judges whether or not an image which is currentlydisplayed clearly appears on the image data captured by the imaging unit130 based on the image data. Then, if the image which is currentlydisplayed clearly appears, it is recognized that the image is clearlydisplayed on the projection screen 110. Further, the display controlunit 141 can recognize positional relationship between the projectionunit 120 and the projection screen 110 based on distance information tothe projection screen 110 obtained by utilizing the autofocus functionof the imaging unit 130 and the image data of the projection screen 110captured by the imaging unit 130, and control the projection unit 120 toproject the image on the projection screen 110 at a size, a shape, anangle, or the like, in accordance with the recognized positionalrelationship.

If the image is clearly displayed on the projection screen 110 in thismanner, the user performs predetermined operation on the image of thescreen projected on the projection screen 110 with the finger or thepointer from the surface side of the projection screen 110 so as to givean input instruction to the computer body 10. How the user performs theoperation is imaged with the imaging unit 130 through the projectionscreen 110, and the obtained image data is transmitted to the imageprocessing unit 132. The image processing unit 132 then performspredetermined image processing on the image data (S5) and transmits theimage data subjected to the image processing to the control unit 140.

The reference data generating unit 142 then generates reference datarelating to the image of the screen projected on the projection screen110 based on the image data obtained by the imaging unit 130 (S6).Specifically, the reference data generating unit 142 acquires positiondata (XY coordinates) of the four corners of the outer frame 230 of thescreen as illustrated in FIG. 4 and sets the acquired position data ofthe four corners of the outer frame 230 as the reference data. Thisreference data is temporarily stored in the storage unit 150.

The image data extracting unit 143 then extracts image data in which thefinger or the pointer with which the user performs operation on theimage of the screen projected on the projection screen 110 exists and inwhich the finger or the pointer is brought into focus in the image dataobtained by the imaging unit 130 (S7). By this means, only image data inwhich the finger or the pointer exists in the vicinity of the projectionscreen 110 is acquired. The image data extracted in this manner istemporarily stored in the storage unit 150. Further, the image dataextracting unit 143 generates time data relating to time at which theimage data is captured for the extracted image data and stores thegenerated time data in the storage unit 150 in association with theimage data.

The position data generating unit 144 then generates position data (XYcoordinates) for specifying a position of the finger or the pointer (forexample, a center position of a tip of the finger or a center positionof a tip portion of the pointer) based on the image data extracted bythe image data extracting unit 143 (S8). The generated position data istemporarily stored in the storage unit 150 in association with the imagedata.

Thereafter, the operation determining unit 145 determines content of theoperation performed with the finger or the pointer based on the imagedata extracted by the image data extracting unit 143 (S9). For example,the operation determining unit 145 determines content of the operationby utilizing the position data and the time data associated with eachpiece of image data for a plurality of pieces of image data extracted bythe image data extracting unit 143. Data relating to the recognizedcontent of the operation performed with the finger or the pointer istemporarily stored in the storage unit 150.

The input control unit 146 then recognizes content of an inputinstruction corresponding to the operation performed with the finger orthe pointer based on the data relating to the content of the operationdetermined by the operation determining unit 145, the position datagenerated by the position data generating unit 144, and the referencedata generated by the reference data generating unit 142 (S10), andcontrols the projection unit 120 so as to project an image of a screenin accordance with the recognized content of the input instruction onthe projection screen 110 (S11).

For example, in the case where the user performs double-tap operation ofan icon on the menu screen with the finger, because the input controlunit 146 can specify an icon which is an operation target based onposition data of the finger, and can specify the operation based on datarelating to content of the operation, the input control unit 146 canrecognize that the input instruction corresponding to the operation ofthis time is an input instruction of displaying a screen relating to theabove-described operated icon. Further, in the case where the userperforms tap operation of a character key image on the keyboard image210 with the finger, because the input control unit 146 can specify theoperated character key based on position data of the finger and canspecify the operation based on data relating to content of theoperation, the input control unit 146 can recognize that the inputinstruction corresponding to the operation of this time is an inputinstruction of a character corresponding to the above-described operatedcharacter key.

After processing in step S11, the control unit 140 judges whether or notan instruction for finishing image projection is received from the user(S12). If an instruction for finishing image projection is received, theinput processing illustrated in FIG. 5 is finished. Meanwhile, if aninstruction for finishing image projection is not received, theprocessing shifts to step S5, and the input processing is continued.Note that the user can give an instruction for finishing imageprojection, for example, through touch operation on the touch pad 30 orthe projection screen 110.

In the image projection device of the present embodiment, the imagingunit has a function of focusing, and the image data extracting unitextracts image data in which the finger or the pointer with which theuser performs operation on the image projected on the projection screenexists and in which the finger or the pointer is brought into focus inthe image data obtained by the imaging unit. Therefore, for example, bythe imaging unit being adjusted so as to come into focus within a rangeof 10 mm in a front-back direction from the projection screen along thevertical direction of the projection screen, because the image dataextracted by the image data extracting unit includes only image data inwhich the finger or the pointer exists in the vicinity of the projectionscreen, the operation determining unit can correctly discern content ofoperation by determining the content of the operation performed with thefinger or the pointer based on the image data extracted by the imagedata extracting unit. The input control unit then recognizes content ofan input instruction corresponding to the operation performed with thefinger or the pointer based on data relating to the content of theoperation performed with the finger or the pointer obtained by beingdetermined by the operation determining unit, the position data of thefinger or the pointer generated by the position data generating unit andthe reference data generated by the reference data generating unit, andcontrols the projection unit to project an image in accordance with therecognized content of the input instruction on the projection screen.Therefore, when the user performs various kinds of touch operation onthe image projected on the projection screen, the image projectiondevice of the present embodiment can correctly recognize an inputinstruction in accordance with the touch operation.

Note that the present invention is not limited to the above-describedembodiment, and various modifications can be made within the scope ofthe gist.

For example, while, in the above-described embodiment, a case has beendescribed where a rear-projection type image projection device whichprojects an image from the back side of the projection screen is used asthe image projection device of the present invention, it is alsopossible to use a front-projection type image projection device whichprojects an image from the surface side of the projection screen. FIG. 8illustrates a schematic configuration diagram of a computer terminal 1 aincluding a front-projection type image projection device 100 a which isa modified example of the present invention. The front-projection typeimage projection device 100 a also includes a projection screen 1100, aprojection unit 120, an imaging unit 130, a control unit (notillustrated), a storage unit (not illustrated) and a table on which ascreen is placed (not illustrated) as illustrated in FIG. 8 as with theimage projection device 100 of the above-described embodiment. Here, asthe projection screen 1100, a front-projection screen for displaying animage by projection light being irradiated from a surface side of theprojection screen is used. Specifically, the front-projection screen ismainly divided into three types of a diffusion type screen, a reflectiontype screen and a regression type screen. The diffusion type screen hascharacteristics that incident light is diffusely reflected in alldirections and has a feature of a very large viewing angle. Thereflection type screen has characteristics that incident light isreflected at a reflection angle which is the same angle as an incidentangle, and the regression type screen has characteristics that reflectedlight returns in the same direction as an incident direction of incidentlight. The projection unit 120 projects an image from the surface sideof the projection screen 1100, and the imaging unit 130 images the imagefrom the surface side of the projection screen 1100 to acquire imagedata. Further, the control unit in the image projection device 100 a inthis modified example includes a display control unit, a reference datagenerating unit, an image data extracting unit, an operation determiningunit, a position data generating unit and an input control unit as withthe control unit in the image projection device 100 of theabove-described embodiment. These units are similar to those in theimage projection device 100 of the above-described embodiment exceptthat the projection unit 120, the imaging unit 130 and the table onwhich the screen is placed are disposed on the surface side of theprojection screen 1100.

In the front-projection type image projection device 100 a, when theuser performs operation on the image projected on the projection screen1100 from the surface side of the projection screen 1100 with the fingeror the pointer, how the user performs the operation is imaged by theimaging unit 130 from the surface side of the projection screen 1100.Here, the imaging unit 130 is adjusted so as to come into focus within arange of 15 mm in a front-back direction from the projection screen 1100along the vertical direction of the projection screen 1100. The inputcontrol unit then recognizes content of an input instructioncorresponding to the operation performed with the finger or the pointerbased on data relating to the operation performed with the finger or thepointer obtained by being determined by the operation determining unit,the position data of the finger or the pointer generated by the positiondata generating unit and the reference data for the screen stored in thestorage unit, and controls the projection unit 120 so as to project animage in accordance with the recognized content of the input instructionon the projection screen 1100. Note that, while the image data obtainedby the imaging unit 130 also includes image data in which a portion ofthe image projected on the projection screen 1100 is hidden by thefinger of the pointer, the operation determining unit can recognizeoperation performed with the finger or the pointer by superimposing animage which is currently projected on the projection screen 1100 on theimage data in which the portion of the image is hidden.

Further, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where theimage projection device includes one projection unit, the imageprojection device may include a plurality of projection units. In thiscase, it is possible to project one screen on the projection screen in adivided manner using the plurality of projection units.

Further, in the above-described embodiment and the modified exampleillustrated in FIG. 8 , the projection unit may project an image inwhich blue light, or the like, is reduced by reducing an amount of lightin the blue or violet region, on the projection screen. By this means,it is possible to display an image which is gentle to the eyes of theuser.

Further, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where theimage projection device includes one imaging unit, the image projectiondevice may include a plurality of imaging units.

In addition, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where theprojection screen is a hard-type single sheet screen, the projectionscreen may be a folding screen, a roll-up screen, or the like.

Further, in the above-described embodiment and the modified exampleillustrated in FIG. 8 , a laser pointer which emits laser light may beused as the pointer. In this case, it is necessary to set content ofoperation performed with the laser pointer in advance. For example, itis set such that operation of lighting twice with laser light meansdouble-tap operation, operation of moving at equal to or lower thanpredetermined speed while lighting with laser light means scrolloperation, operation of moving at equal to or higher than predeterminedspeed while lighting with laser light means flick operation, andoperation of moving after lighting a predetermined number of times withlaser light means drag operation. Specifically, the image dataextracting unit recognizes a shape and/or color of laser light from thelaser pointer in the image data obtained by the imaging unit when theuser performs operation on the image projected on the projection screenby irradiating laser light from the laser pointer on the projectionscreen, and acquires image data in which the recognized shape and/orcolor of the laser light exists as image data in which the finger or thepointer with which the user performs operation on the image exists.Here, concerning recognition of color of the laser light, for example, afrequency of color of the laser light is preset, and in the case wherethe image data extracting unit recognizes the preset color, it may beregarded that the laser light exists. While there can be a case wherecolor similar to the preset color exists in the image, by using color ofa frequency in red region which is not included in color used in anormal presentation screen, or the like, as color of the laser light,the color of the laser light is not erroneously recognized. Therefore,also in the case where a laser pointer is used as the pointer, the imageprojection device of the present invention can correctly recognize aninput instruction from the user. Note that a cross-sectional shape oflaser light emitted from the laser pointer is not limited to a circle,and may be a star shape, a cross shape, a triangle, or the like. By thismeans, the image data extracting unit can easily perform processing ofjudging existence of the pointer.

Further, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where theprojection unit and the imaging unit are provided within a housing ofthe computer body, the projection unit and the imaging unit may beprovided within a housing different from the housing of the computerbody. In this case, by a communication unit and a communication controlunit being provided at each of the housing in which the projection unitand the imaging unit are provided and the housing of the computer body,it is possible to perform wireless data communication between theprojection unit and the imaging unit, and the control unit. Typically,the projection unit and/or the imaging unit may be provided within ahousing different from a housing in which the control unit is provided.

Further, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where a frameis pictured in each image projected on the projection screen by theprojection unit, it is not always necessary that a frame is pictured ineach image projected on the projection screen by the projection unit.Further, instead of a frame being pictured in each image, for example,predetermined marks may be pictured at four corners of each imageprojected on the projection screen by the projection unit. In this case,the reference data generating unit recognizes positions of the marks inthe image in the imaging range based on the image data obtained by theimaging unit and sets data relating to the recognized positions of themarks as reference data.

Further, in the image projection device in the above-describedembodiment and the modified example illustrated in FIG. 8 , it ispossible to omit a table on which a screen is placed and a liquidcrystal display unit. Still further, while a case has been described inthe above-described embodiment where the imaging unit is adjusted so asto come into focus within a range of 10 mm in a front-back directionfrom the projection screen along the vertical direction of theprojection screen, and a case has been described in the modified exampleillustrated in FIG. 8 where the imaging unit is adjusted so as to comeinto focus within a range of 15 mm in a front-back direction from theprojection screen along the vertical direction of the projection screen,a range in which the imaging unit comes into focus is not limited to arange of 10 mm or 15 mm in a front-back direction from the projectionscreen along the vertical direction of the projection screen, and maybe, for example, a range of approximately 50 mm in a front-backdirection from the projection screen along the vertical direction of theprojection screen. The imaging unit is adjusted so as to come into focuswithin a range of a predetermined distance in a front-back directionfrom the projection screen along the vertical direction of theprojection screen. Typically, if the predetermined distance is small,while accuracy for discerning content of operation is improved, itbecomes difficult for a user to perform operation. Meanwhile, if thepredetermined distance is large, while operability improves for theuser, accuracy for discerning content of operation decreases. In orderto improve accuracy for discerning content of operation and improveoperability, it is preferable that the above-described predetermineddistance is approximately between 5 mm to 25 mm in a case of arear-projection type image projection device and approximately between10 mm and 50 mm in a case of a front-projection type image projectiondevice. In the rear-projection type image projection device, because theimaging unit images an image from a back side of the projection screen,the image data extracting unit needs to extract image data in which thefinger or the pointer with which operation is performed on theprojection screen exists and in which a surface of the finger or thepointer on a side facing the projection screen is brought into focus. Inthis case, for example, if the user performs tap operation with theindex finger, it is considered that a finger pad in the index fingermoves from the surface of the projection screen by approximately between5 mm and 25 mm. Therefore, in the rear-projection type image projectiondevice, it is preferable that the above-described predetermined distanceis approximately between 5 mm and 25 mm. In contrast to this, in thefront-projection type image projection device, because the imaging unitimages an image from the surface side of the projection screen, theimage data extracting unit needs to extract image data in which thefinger or the pointer with which operation is performed on theprojection screen exists and in which a surface of the finger or thepointer on a side opposite to a side facing the projection screen isbrought into focus. For example, in the case where the user performs tapoperation with the index finger, given that a thickness of the finger ata nail portion of the index finger is approximately between 5 mm and 15mm, the above-described predetermined distance is only required to beset at approximately between 10 mm and 40 mm. Further, in thefront-projection type image projection device, because the finger, orthe like, with which operation is performed is located on the same sideas the side of the projection unit, it is considered that accuracy fordiscerning content of operation decreases compared to therear-projection type image projection device. Therefore, in thefront-projection type image projection device, there is a case where itwould be better for the user to perform operation with overaction usingthe finger, or the like. Given that operation is performed withoveraction in such a manner, it is preferable that the above-describedpredetermined distance is approximately 50 mm. In addition, in the casewhere a size of the projection screen is large, and, for example, a longpointing stick is used as the pointer, in both the rear-projection typeimage projection device and the front-projection type image projectiondevice, it is preferable that the above-described predetermined distanceis approximately 100 mm.

Further, while, in the above-described embodiment and the modifiedexample illustrated in FIG. 8 , a case has been described where theimaging unit is adjusted to come into focus within a range of apredetermined distance in a front-back direction from the projectionscreen along the vertical direction of the projection screen, a centerof the range in which the imaging unit comes into focus does notnecessarily have to be located on the projection screen. For example, inthe rear-projection type image projection device, a position away fromthe projection screen to a rear side by a distance x/2 along thevertical direction of the projection screen is set as the center of therange in which the imaging unit comes into focus, and the imaging unitmay be adjusted so as to come into focus within a range of the distancex/2 in a front-back direction from the center along the verticaldirection of the projection screen. That is, in this case, the range inwhich the imaging unit comes into focus becomes a range of a distance xfrom the projection screen to a rear side along the vertical directionof the projection screen. Further, in the front-projection type imageprojection device, a position away from the projection screen to a frontside by a distance x/2 along the vertical direction of the projectionscreen is set as the center of the range in which the imaging unit comesinto focus, and the imaging unit may be adjusted so as to come intofocus within a range of the distance x/2 in a front-back direction fromthe center along the vertical direction of the projection screen. Thatis, in this case, the range in which the imaging unit comes into focusbecomes a range within the distance x from the projection screen to afront side along the vertical direction of the projection screen.

While, in the above-described embodiment and the modified exampleillustrated in FIG. 8 , a case has been described where the imageprojection device of the present invention is incorporated into thecomputer terminal, the image projection device of the present inventioncan be incorporated into various kinds of equipment. For example, theimage projection device of the present invention can be provided at amobile terminal such as a smartphone and a mobile phone, a wristwatch, acar, a television, or the like.

FIG. 9A is a schematic perspective view of a smartphone in which theimage projection device of the present invention is provided, and FIG.9B is a schematic side view of the smartphone. In the case where theimage projection device of the present invention is applied to asmartphone 300, the projection unit 120 and the imaging unit 130 areprovided at an upper end portion of the smartphone 300 so as to beadjacent to each other as illustrated in FIG. 9A, and the projectionscreen 110 is attached at a lower portion of the smartphone 300 in astate where the projection screen 110 is tilted as illustrated in FIG.9B. The projection screen 110 is a rear projection screen for displayingan image by projection light being irradiated from a back side of theprojection screen 110. Further, the control unit and the storage unit ofthe image projection device of the present invention are incorporatedinto the smartphone 300. When the projection unit 120 displays an imageof a menu screen from the back side of the projection screen 110, theuser gives an input instruction by performing operation on the menuscreen from the surface side of the projection screen 110 with thefinger or the pointer.

FIG. 10A is a schematic perspective view of a wristwatch in which theimage projection device of the present invention is provided, and FIG.10B is a diagram illustrating how a projection screen is provided at thewristwatch. In FIG. 10 , a body portion of the wristwatch 400 isdisplayed, and a watchband is omitted. The wristwatch 400 to which theimage projection device of the present invention is applied has afunction as an information terminal. The control unit and the storageunit of the image projection device of the present invention areincorporated into the wristwatch 400. Further, the projection unit 120and the imaging unit 130 are provided at an upper end portion on thesurface of the wristwatch 400 as illustrated in FIG. 10A. The projectionscreen 110 a is a roll-type projection screen, and stored in a screenstorage unit 170. The screen storage unit 170 is provided at a left endportion of the wristwatch 400. At a right end portion of the wristwatch400, hooks 180, 180 for hanging the projection screen 110 a areprovided. When the image projection device of the present invention isused, as illustrated in FIG. 10B, the user draws the projection screen110 a from the screen storage unit 170 and hangs the projection screen110 a on the hooks 180, 180, so that the projection screen 110 a isplaced so as to cover a surface of the wristwatch 400. When theprojection unit 120 displays an image of a menu screen from the backside of the projection screen 110 a, the user gives an input instructionby performing operation on the menu screen from a surface side of theprojection screen 110 a with the finger or the pointer.

Further, there are various application methods in the case where theimage projection device of the present invention is applied to a car.For example, there can be a method in which the projection unit, theimaging unit, the control unit and the storage unit in the imageprojection device of the present invention are incorporated into anengine room, or the like, of the car, and the projection unit projects ascreen to be projected on a windshield of a car body (projectionscreen), a method in which the projection unit, the imaging unit, thecontrol unit and the storage unit in the image projection device of thepresent invention are incorporated into a dashboard portion of the car,and the projection unit projects a screen to be projected on a controlpanel (projection screen), or the like. In these methods, the imageprojection device of the present invention is used as a rear-projectiontype image projection device. FIG. 11A illustrates an example of a carin which the rear-projection type image projection device is provided.Further, there can be a method in which the projection unit, the imagingunit, the control unit and the storage unit in the image projectiondevice of the present invention are incorporated into a dashboardportion, an interior ceiling portion, or the like, of the car, and theprojection unit projects a screen to be projected on a windshield of acar body (projection screen). In this method, the image projectiondevice of the present invention is used as a front-projection type imageprojection device. FIG. 11B illustrates an example of a car in which thefront-projection type image projection device of the present inventionis provided.

Further, the image projection device of the present invention can beincorporated into even a device such as a desk or a table used in dailylife, in the case where the device has a function as a terminal. FIG. 12illustrates an example of a desk 500 in which the image projectiondevice of the present invention is provided. In this case, theprojection screen 110 is provided on a top board 510 of the desk 500,and the projection unit, the imaging unit, the control unit and thestorage unit are disposed at a lower side of the top board 510. Then,FIG. 13 illustrates an example of a compact mirror 600 in which theimage projection device of the present invention is provided. Here, FIG.13A is a schematic perspective view of the compact mirror when themirror is used, FIG. 13B is a schematic side view of the compact mirrorwhen the mirror is used, FIG. 13C is a schematic perspective view of thecompact mirror when an image is projected, and FIG. 13D is a schematicside view of the compact mirror when the image is projected. The compactmirror 600 has a function as an information terminal, and, asillustrated in FIG. 13 , includes an upper cover 610, a lower cover 620,a mirror 630 and the image projection device of the present invention.The projection unit 120, the imaging unit 130, the control unit and thestorage unit of the image projection device of the present invention areincorporated into the upper cover 610. As illustrated in FIG. 13A andFIG. 13B, the projection screen 110 is disposed on a back side of themirror 630. As illustrated in FIG. 13C and FIG. 13D, by the mirror 630being tilted forward and the projection screen 110 being separated fromthe upper cover 610, the image projection device of the presentinvention can be used.

The term module (and other similar terms such as unit, subunit,submodule, etc.) in this disclosure may refer to a software module, ahardware module, or a combination thereof. A software module (e.g.,computer program) may be developed using a computer programminglanguage. A hardware module may be implemented using processingcircuitry and/or memory. Each module can be implemented using one ormore processors (or processors and memory). Likewise, a processor (orprocessors and memory) can be used to implement one or more modules.Moreover, each module can be part of an overall module that includes thefunctionalities of the module.

INDUSTRIAL APPLICABILITY

As described above, in the image projection device of the presentinvention, the imaging unit has a function of focusing, and the imagedata extracting unit extracts image data in which the finger or thepointer with which the user performs operation on the image projected onthe projection screen exists and in which the finger or the pointer isbrought into focus, in the image data obtained by the imaging unit.Therefore, for example, by the imaging unit being adjusted so as to comeinto focus within a range of a predetermined short distance in afront-back direction from the projection screen along the verticaldirection of the projection screen, because the image data extracted bythe image data extracting unit includes only image data in which thefinger or the pointer exists in the vicinity of the projection screen,the operation determining unit can correctly discern content ofoperation by determining the content of the operation performed with thefinger or the pointer based on the image data extracted by the imagedata extracting unit. Therefore, the image projection device of thepresent invention is preferably used while being incorporated intovarious kinds of equipment, machine or devices such as a computerterminal, a mobile terminal, a wristwatch, a car and a television.

DESCRIPTION OF THE REFERENCE NUMERAL

-   1, 1 a computer terminal-   10 computer body-   20 liquid crystal display unit-   30 touch panel-   100, 100 a image projection device-   110, 110 a, 1100 projection screen-   120 projection unit-   130 imaging unit-   131 camera unit-   132 image processing unit-   133 camera control unit-   133 a autofocus control unit-   140 control unit-   141 display control unit-   142 reference data generating unit-   143 image data extracting unit-   144 position data generating unit-   145 operation determining unit-   146 input control unit-   150 storage unit-   160 table on which screen is placed-   161 a, 161 b base portion-   162 pole portion-   163 concave portion-   170 screen storage unit-   180 hook-   200 character input screen-   210 keyboard image-   220 display region-   230 outer frame-   300 smartphone-   400 wristwatch-   500 desk-   510 top board-   600 compact mirror-   610 upper cover-   620 lower cover-   630 mirror

1. An image projection device comprising: a transmission type projectionscreen configured to: display an image by projection light beingirradiated from a back side of the projection screen and a projectionunit configured to project and display a predetermined image on theprojection screen from the back side of the projection screen, recognizecontent of an input instruction corresponding to touch operation when auser performs tap operation or double-tap operation as the touchoperation on an image projected on the projection screen from a surfaceside of the projection screen using a finger of the user or a pointer,and project an image in accordance with the recognized content of theinput instruction on the projection screen; an imaging unit configuredto: image the image projected on the projection screen, and acquireimage data focused within a range of a predetermined distance from theprojection screen to the surface side of the projection screen along avertical direction of the projection screen, wherein the predetermineddistance is between 5 mm and 100 mm; a reference data generating unitconfigured to generate reference data specifying a position and a sizeof the image projected on the projection screen in an imaging range ofthe imaging unit based on the image data acquired by the imaging unit;an image data extracting unit configured to: extract image data in whicha finger or a pointer with which a user performs operation on the imageprojected on the projection screen exists and in which the finger or thepointer is focused in the image data obtained by the imaging unit, andassociate time data relating to time at which the image data is capturedwith the image data, wherein the image data extracted by the image dataextracting unit includes only image data in which the finger or thepointer exists within the predetermined distance from the projectionscreen along a vertical direction of the projection screen; a positiondata generating unit configured to: generate position data forspecifying a position of the finger or the pointer in the imaging rangeof the imaging unit based on the image data extracted by the image dataextracting unit, and associate the generated position data with theimage data; an operation determining unit configured to determinecontent of operation performed with the finger or the pointer based onthe position data and the time data associated with each piece of imagedata for a series of pieces of image data extracted in chronologicalorder by the image data extracting unit; and an input control unitconfigured to: recognize content of an input instruction correspondingto the operation performed with the finger or the pointer based on: thecontent of the operation determined by the operation determining unit,the position data generated by the position data generating unit, andthe reference data generated by the reference data generating unit, andcontrol the projection unit to project an image in accordance with therecognized content of the input instruction on the projection screen,wherein in a case where the operation determining unit recognizes, afterimage data is extracted in which the position of the finger or thepointer remains virtually unchanged a predetermined period, image datais extracted where the finger or the pointer exists at substantially thesame position for the predetermined period again before a certain periodof time has elapsed, the operation determining unit is configured todetermine that the double-tap operation is performed with the finger orthe pointer, and in a case where the operation determining unitrecognizes, after image data is extracted in which the position of thefinger or the pointer remains virtually unchanged for the predeterminedperiod, image data is not extracted where the finger or the pointerexists at substantially the same position before the period of time haselapsed, the operation determining unit is configured to determine thatthe tap operation is performed with the finger or the pointer.
 2. Theimage projection device according to claim 1, wherein a frame ispictured in each image projected on the projection screen orpredetermined marks are pictured at four corners of each image projectedon the projection screen, and the reference data generating unit isfurther configured to recognize a position of the frame or the marks ofthe image in an imaging range of the imaging unit based on the imagedata obtained by the imaging unit and sets data relating to therecognized position of the frame or the marks as reference data.
 3. Theimage projection device according to claim 1, wherein when the userperforms operation on the image from a surface side of the projectionscreen with a finger or a pointer, the image data extracting unit isconfigured to: recognize a shape, or a shape and color corresponding tothe finger or the pointer in the image data captured through theprojection screen, and acquire image data in which the recognized shapeor shape and color exists as image data in which the finger or thepointer with which the user performs operation on the image exists. 4.The image projection device according to claim 1, wherein when the userperform operation on the image while bringing a finger or a pointer intocontact with the image from a surface side of the projection screen, theimage data extracting unit is configured to: recognize change of ashape, change of color or change of a shape and color of the finger orthe pointer when the finger or the pointer is brought into contact withthe projection screen in the image data captured through the projectionscreen, and acquire image data in which the recognized change of ashape, change of color, or change of a shape and color exists as imagedata in which the finger or the pointer with which the user performsoperation on the image exists.
 5. The image projection device accordingto claim 1, wherein in a case where a laser pointer which emits laserlight is used as the pointer, when the user performs operation on theimage projected on the projection screen by irradiating laser light fromthe laser pointer on the projection screen, the image data extractingunit is configured to: recognize a shape and/or color of the laser lightfrom the laser pointer in the image data obtained by the imaging unit,and acquire image data in which the recognized shape and/or color of thelaser light exists as image data in which the finger or the pointer withwhich the user performs operation on the image exists.
 6. The imageprojection device according to claim 1, further comprising: a table onwhich a screen is disposed for attaching the projection screen so that adistance between the projection screen and the projection unit and adistance between the projection screen and the imaging unit arerespectively maintained at substantially fixed distances.
 7. The imageprojection device according to claim 1, wherein the reference datagenerating unit, the image data extracting unit, the position datagenerating unit, the operation determining unit and the input controlunit are provided within one housing, and the projection unit and/or theimaging unit are provided within a housing different from the housing.8. A computer terminal, a wristwatch, a mobile terminal or a carcomprising the image projection device according to claim
 1. 9. Theimage projection device according to claim 1, wherein the range whichcan be imaged by the imaging unit is the whole projection screen.
 10. Animage projection device comprising: a projection screen configured todisplay an image by projection light being irradiated from a surfaceside of the projection screen; and a projection unit configured to:project and display a predetermined image on the projection screen fromthe surface side of the projection screen, recognize content of an inputinstruction corresponding to touch operation when a user performs tapoperation or double-tap operation as the touch operation on an imageprojected on the projection screen from the surface side of theprojection screen using a finger of the user or a pointer, and projectan image in accordance with the recognized content of the inputinstruction on the projection screen; an imaging unit configured to:image the image projected on the projection screen, and acquire imagedata focused within a range of a predetermined distance from theprojection screen to the surface side of the projection screen along avertical direction of the projection screen, wherein the predetermineddistance is between 5 mm and 100 mm; a reference data generating unitconfigured to generate reference data for specifying a position and asize of the image projected on the projection screen in an imaging rangeof the imaging unit based on the image data acquired by the imagingunit; an image data extracting unit configured to: extract image data inwhich a finger or a pointer with which a user performs operation on theimage projected on the projection screen exists and in which the fingeror the pointer is brought into focus in the image data obtained by theimaging unit, and associate time data relating to time at which theimage data is captured with the image data, wherein the image dataextracted by the image data extracting unit includes only image data inwhich the finger or the pointer exists within the predetermined distancefrom the projection screen along a vertical direction of the projectionscreen; a position data generating unit configured to: generate positiondata for specifying a position of the finger or the pointer in theimaging range of the imaging unit based on the image data extracted bythe image data extracting unit, and associate the generated positiondata with the image data; an operation determining unit configured todetermine content of operation performed with the finger or the pointerbased on the position data and the time data associated with each pieceof image data for a series of pieces of image data extracted inchronological order by the image data extracting unit; and an inputcontrol unit configured to: recognize content of an input instructioncorresponding to the operation performed with the finger or the pointerbased on: the content of the operation determined by the operationdetermining unit, the position data generated by the position datagenerating unit, and the reference data generated by the reference datagenerating unit, and control the projection unit to project an image inaccordance with the recognized content of the input instruction on theprojection screen, wherein in a case where the operation determiningunit recognizes, after image data in which the position of the finger orthe pointer remains virtually unchanged or a predetermined period, imagedata is extracted where the finger or the pointer exists atsubstantially the same position is extracted for the predeterminedperiod again before a certain period of time has elapsed, the operationdetermining unit is configured to determine that the double-tapoperation is performed with the finger or the pointer, and in a casewhere the operation determining unit recognizes, after image data inwhich the position of the finger or the pointer remains virtuallyunchanged is extracted for the predetermined period, image data is notextracted where the finger or the pointer exists at substantially thesame position before the period of time has elapsed, the operationdetermining unit is configured to determine that the tap operation isperformed with the finger or the pointer.
 11. The image projectiondevice according to claim 10, wherein a frame is pictured in each imageprojected on the projection screen or predetermined marks are picturedat four corners of each image projected on the projection screen, thereference data generating unit is configured to recognize a position ofthe frame or the marks of the image in an imaging range of the imagingunit based on the image data obtained by the imaging unit and sets datarelating to the recognized position of the frame or the marks asreference data.
 12. The image projection device according to claim 10,wherein in a case where a laser pointer which emits laser light is usedas the pointer, when the user performs operation on the image projectedon the projection screen by irradiating laser light from the laserpointer on the projection screen, the image data extracting unit isconfigured to: recognize a shape and/or color of the laser light fromthe laser pointer in the image data obtained by the imaging unit, andacquire image data in which the recognized shape and/or color of thelaser light exists as image data in which the finger or the pointer withwhich the user performs operation on the image exists.
 13. The imageprojection device according to claim 10, wherein the reference datagenerating unit, the image data extracting unit, the position datagenerating unit, the operation determining unit and the input controlunit are provided within one housing, and the projection unit and/or theimaging unit are provided within a housing different from the housing.14. A computer terminal, a wristwatch, a mobile terminal or a carcomprising the image projection device according to claim
 10. 15. Theimage projection device according to claim 10, wherein the range whichcan be imaged by the imaging unit is the whole projection screen. 16.The image projection device according to claim 1, wherein the image dataextracting unit is further configured to: perform processing ofexcluding data concerning an image which is currently projected on theprojection screen on the image data obtained by the imaging unit, andjudge whether the finger or the pointer exists based on the image datasubjected to the processing.
 17. The image projection device accordingto claim 10, wherein the image data extracting unit is furtherconfigured to: perform processing of excluding data concerning an imagewhich is currently projected on the projection screen on the image dataobtained by the imaging unit, and judge whether the finger or thepointer exists based on the image data subjected to the processing. 18.The image projection device according to claim 1, wherein the imagingunit further comprises: a visible light camera configured to focus anddisposed so that the optical axis of the visible light camera and thenormal direction of the projection screen are substantially parallel toeach other.
 19. The image projection device according to claim 10,wherein the imaging unit further comprises: a visible light cameraconfigured to focus and disposed so that the optical axis of the visiblelight camera and the normal direction of the projection screen aresubstantially parallel to each other.